RESTful Services
Overview
There are a large number of information resources useful to applications beyond traditional databases. Many of these information resources are Internet accessible. Representational state transfer (REST) is a common pattern for interacting with these resources. A discussion on REST can be found at https://en.wikipedia.org/wiki/Representational_state_transfer. In general there is a known URL which may accept additional parameters to obtain some well known data format.
Qarbine can interact with a RESTful endpoint to obtain complex records such as shown below.
Qarbine can easily analyze this deeply nested data and format an analysis report. This interaction can also be embedded into applications for a seamless end user experience. Sample output is shown below.
This report result can then be exported into various popular formats and easily shared within leading collaboration tools.
Sample RESTful Service
Here is an example of a RESTful endpoint that returns a list of phone accessories
This example endpoint requires a “data” parameter which is the name of a data file. Sample results are shown below.
Defining a Data Source
Overview
A Data Source is a Qarbine component responsible for retrieving data from somewhere. At a high level it has a name, a description and some arbitrary query string which when sent to the associated Qarbine Data Service endpoint returns some data. The overall execution flow for an analysis, including the optional prompt component, is shown below.
A single data source can be referenced by name from multiple Qarbine template components. This enables a single point of change when perhaps, an index is added, or some other query tweak is necessary. The alternative is to attempt to find all templates impacted by a schema or index change for example. The component reusability is especially beneficial when team members have varying roles and skills.
Query Language
The general format for the query specification is a list of optional lines.
set …
SELECT * FROM restResult WHERE …
The recognized SET line patterns are listed below:
- set url URL
- set method GET | POST
- set authenticationBasic TOKEN
- set urlArgument KEY=VALUE
- set header HEADER_KEY=HEADER_VALUE
The “set url …” line overrides the server template value in the data service configuration. This is permitted if the data services’ driver option “allowOtherUrls” is true.
There can be multiple urlArgument lines. The line’s value argument is automatically URL encoded. These are concatenated to form the final URL data argument.
As with other Qarbine query specifications, any line starting with ‘#’ or ‘//’ is treated as a comment.
The optional SELECT line further filters the results from the RESTful service’s reply.
The response from the RESTful service is expected to be a string representation of a list of elements. The format may be CSV, TSV, or JSON. The SELECT table name (restResult above) provides guidance on the format. If the SELECT table name has no file extension then a probe is done on the raw string reply looking for JSON '{' and ']' and the tab character with the default being CSV. A table of “restResult.csv” indicates to expect a CSV formatted reply.
Example
Shown below is an example query
Sample results are shown below.
The details of one of the elements are shown below.
A more detailed analysis can be seen by clicking the Schema Analysis tab.
Sample output is shown below.
 | The structure content below “limits” is,  |
This Data Source can be saved with the name “Phone accessories”.
You can quickly create a starting analysis template by clicking on the Create Components 
.
. . .
. . .
In this case the generated template references the saved Data Source. Sample template run results are shown below.The template quickly provides the information in a roughly formatted manner. It offers a good starting for additional formatting tweaks.
Shown below is another example query
Sample results are shown below.
Qarbine pragmas can be applied to trim or otherwise manipulate the answer set. For example
#pragma deleteFields _id
set urlArgument data=phone.json
select * from restResult where price > 300
Sample results are shown below.
Managing Answer Set Size
The default maximum number of rows starts off at 25 for a new data source. This is useful to evolve a query from a concept to one that you have verified returns the desired answer set. As noted, any native way of limiting an answer set size is the preferred approach. This setting is in the component dialog as shown below and also accessible by clicking the ‘Gear’ icon.
Once you are done drafting you can adjust this parameter. A “0” indicates there is no maximum. A number greater than 0 indicates to limit the final answer set size to that number of rows. This answer set truncation comes after any native query limit. So, if the answer set from the data endpoint is quite large, that content has to be returned to the Qarbine host. It then may truncate the number of rows. It is best to truncate at the query level (i.e., use a limit) to reduce the content sent from the data endpoint to the Qarbine host in the first place.
Adjusting the Maximum Rows
Recall the default maximum rows at the component level is 25. When you are satisfied with your query you can change that setting by clicking.
Adjust the setting to “0” indicating no Qarbine answer set truncation.
Click
Defining an Analysis Template
Overview
A template defines how to process the data being retrieved from Data Source queries and other data expressions. It also defines formulas, formatting options, and other analysis and presentation options. Team members can define templates which can be easily discovered by others for their running or to use as a starting point for other templates. The overall execution flow for an analysis, including the optional prompt component, is shown below
Using the Template Designer
The following template is located at “example/Other data types/REST/Phone accessories enhanced”.
We can load the generated template from above. Update the name by accessing the properties dialog by clicking 
. The name is updated in the dialog. Also note the data source reference has been all set as well.
. . .
Save the new name by clicking
Next, we can make some formatting adjustments and in just a few minutes have output such as the following.
To save the updated template click 
.
Using the doRest() Macro Function
An alternative to defining a Data Service is to use the doRest() macro function. Its signature is shown below.
authorization, cookie, returnReplyField)
| Argument | Description |
|---|---|
| url | The HTTP or HTTPS endpoint. |
| method | Either GET or POST. |
| encodedArgs | The arguments for the request. When the method is GET they are appended to the URL value. FOr POST they are the request’s body payload. |
| replyDataType | Use ‘JSON’ to create a structured JSON object. |
| authorization | Any authorization header value which is formatted as “basic VALUE’ |
| cookie | Any cookie value |
| returnReplyField | Use this in conjunction with the JSON replyDataType when the value of interest is nested within the root object.. |
A sample endpoint URL is https://www.7timer.info/bin/astro.php?lon=113.2&lat=23.1&ac=0& unit=metric&output=json&tzshift=0
It returns a JSON object with the following structure.
Another endpoint URL example is
https://reqres.in/api/users/page=2
It returns a JSON object with the following structure.
Here is an example of running that endpoint in the Data Source Designer and using the “Formula” Data Service.
Next Steps
Accessing Your Data
To configure access to your REST endpoint see the guides at
Querying Your Data
For database specific interaction guides navigate to
References
Useful links include